Skip to main content

Focused Crawls

Internet Archive

Focused crawls are collections of frequently-updated webcrawl data from narrow (as opposed to broad or wide) web crawls, often focused on a single domain or subdomain.



rss RSS

Show sorted alphabetically
Show sorted alphabetically
SHOW DETAILS
up-solid down-solid
Prior Page
eye
Title
Date Archived
Creator
weather.com
weather.com
collection
685
ITEMS
10.9M
VIEWS
by http://weather.com/
collection
eye 10.9M
theguardian.com
theguardian.com
collection
1,123
ITEMS
11.5M
VIEWS
by theguardian.com
collection
eye 11.5M
wordpress.org
wordpress.org
collection
1,335
ITEMS
16.3M
VIEWS
by http://wordpress.org
collection
eye 16.3M
top_domains-08500
top_domains-08500
collection
853
ITEMS
17.9M
VIEWS
by Internet Archive
collection
eye 17.9M
cnn.com
cnn.com
collection
631
ITEMS
13.1M
VIEWS
by http://cnn.com
collection
eye 13.1M
liveinternet.ru
liveinternet.ru
collection
749
ITEMS
12.4M
VIEWS
by liveinternet.ru
collection
eye 12.4M
npr.org
npr.org
collection
3,634
ITEMS
12.4M
VIEWS
by http://npr.org/
collection
eye 12.4M
reddit.com
reddit.com
collection
301
ITEMS
12.7M
VIEWS
by http://reddit.com
collection
eye 12.7M
current.com
current.com
collection
337
ITEMS
17M
VIEWS
by http://current.com/
collection
eye 17M
pbs.org
pbs.org
collection
4,933
ITEMS
10.4M
VIEWS
by http://pbs.org/newshour/
collection
eye 10.4M
flickr.com
flickr.com
collection
1,527
ITEMS
13.1M
VIEWS
by http://flickr.com
collection
eye 13.1M
newser.com
newser.com
collection
249
ITEMS
15M
VIEWS
by http://newser.com/
collection
eye 15M
seattlepi.com
seattlepi.com
collection
1,470
ITEMS
10M
VIEWS
by http://www.seattlepi.com/
collection
eye 10M
youtube.com
youtube.com
collection
185
ITEMS
12.7M
VIEWS
by http://youtube.com
collection
eye 12.7M
pinterest.com
pinterest.com
collection
255
ITEMS
16.9M
VIEWS
by pinterest.com
collection
eye 16.9M
gulfnews.com
gulfnews.com
collection
379
ITEMS
7.4M
VIEWS
by http://gulfnews.com/
collection
eye 7.4M
edition.cnn.com
edition.cnn.com
collection
602
ITEMS
11.1M
VIEWS
by http://edition.cnn.com/
collection
eye 11.1M
tumblr.com
tumblr.com
collection
447
ITEMS
17.9M
VIEWS
by http://tumblr.com
collection
eye 17.9M
indiatimes.com
indiatimes.com
collection
431
ITEMS
6.6M
VIEWS
by http://indiatimes.com
collection
eye 6.6M
cbc.ca
cbc.ca
collection
881
ITEMS
13.3M
VIEWS
by http://cbc.ca/news/
collection
eye 13.3M
lefigaro.fr
lefigaro.fr
collection
1,813
ITEMS
7.6M
VIEWS
by http://lefigaro.fr/
collection
eye 7.6M
abc.net.au-news
abc.net.au-news
collection
1,109
ITEMS
5.7M
VIEWS
by http://abc.net.au/news/
collection
eye 5.7M
usatoday.com
usatoday.com
collection
1,035
ITEMS
9.5M
VIEWS
by http://usatoday.com
collection
eye 9.5M
usnews.com
usnews.com
collection
211
ITEMS
6.6M
VIEWS
by http://usnews.com/
collection
eye 6.6M
msn.com
msn.com
collection
1,179
ITEMS
8.8M
VIEWS
by http://msn.com
collection
eye 8.8M
fbdailycrawl.io
fbdailycrawl.io
collection
1,805
ITEMS
3.9M
VIEWS
by Internet Archive
collection
eye 3.9M
espn.go.com
espn.go.com
collection
1,888
ITEMS
10.7M
VIEWS
by http://espn.go.com
collection
eye 10.7M
wix.com
wix.com
collection
453
ITEMS
6.7M
VIEWS
by wix.com
collection
eye 6.7M
kp.ru
kp.ru
collection
595
ITEMS
9M
VIEWS
by http://kp.ru/
collection
eye 9M
elmundo.es
elmundo.es
collection
595
ITEMS
8.3M
VIEWS
by http://elmundo.es/
collection
eye 8.3M
telegraph.co.uk
telegraph.co.uk
collection
213
ITEMS
9.2M
VIEWS
by http://telegraph.co.uk
collection
eye 9.2M
ehow.com
ehow.com
collection
295
ITEMS
8.4M
VIEWS
by ehow.com
collection
eye 8.4M
asahi.com
asahi.com
collection
513
ITEMS
8.2M
VIEWS
by http://www.asahi.com/
collection
eye 8.2M
timesofindia.indiatimes.com
timesofindia.indiatimes.com
collection
431
ITEMS
5.3M
VIEWS
by http://timesofindia.indiatimes.com/
collection
eye 5.3M
dr1.com
dr1.com
collection
127
ITEMS
10.8M
VIEWS
by http://dr1.com/
collection
eye 10.8M
newsweek.com
newsweek.com
collection
612
ITEMS
9.3M
VIEWS
by http://newsweek.com/
collection
eye 9.3M
example.com
example.com
collection
1,366
ITEMS
3.9M
VIEWS
by https://example.com
collection
eye 3.9M
This is not an archive of example.com. This is a "focused crawl" (H3 deriver module) test using a "seedsfile" CXML template with a large (5-10K) seed list to determine how much we can crawl in a day, and to investigate limitations, constraints, or bottlenecks in this context.
us.cnn.com
us.cnn.com
collection
596
ITEMS
9M
VIEWS
by http://us.cnn.com/
collection
eye 9M
propublica.org
propublica.org
collection
961
ITEMS
7.3M
VIEWS
by http://propublica.org/
collection
eye 7.3M
ndtv.com
ndtv.com
collection
1,167
ITEMS
5.9M
VIEWS
by http://ndtv.com/
collection
eye 5.9M
wsj.com
wsj.com
collection
710
ITEMS
6.7M
VIEWS
by http://wsj.com
collection
eye 6.7M
lemonde.fr
lemonde.fr
collection
576
ITEMS
5.3M
VIEWS
by http://lemonde.fr/
collection
eye 5.3M
cbsnews.com
cbsnews.com
collection
1,566
ITEMS
7.4M
VIEWS
by http://cbsnews.com
collection
eye 7.4M
soundcloud.com
soundcloud.com
collection
307
ITEMS
7.3M
VIEWS
by soundcloud.com
collection
eye 7.3M
chinadigitaltimes.net
chinadigitaltimes.net
collection
523
ITEMS
7.3M
VIEWS
by http://chinadigitaltimes.net/
collection
eye 7.3M
online.wsj.com-home-page
online.wsj.com-home-page
collection
659
ITEMS
6.3M
VIEWS
by http://online.wsj.com/home-page
collection
eye 6.3M
sify.com
sify.com
collection
154
ITEMS
7.3M
VIEWS
by http://sify.com/
collection
eye 7.3M
wordpress.com
wordpress.com
collection
907
ITEMS
7.1M
VIEWS
by http://wordpress.com
collection
eye 7.1M
rtl.de
rtl.de
collection
1,569
ITEMS
5M
VIEWS
by http://rtl.de/
collection
eye 5M
nikkei.com
nikkei.com
collection
341
ITEMS
4.4M
VIEWS
by http://nikkei.com/
collection
eye 4.4M
thecostaricanews.com
thecostaricanews.com
collection
225
ITEMS
8.3M
VIEWS
by http://thecostaricanews.com/
collection
eye 8.3M
rfi.fr
rfi.fr
collection
2,727
ITEMS
7.5M
VIEWS
by http://rfi.fr/
collection
eye 7.5M
pornhub.com
pornhub.com
collection
1,053
ITEMS
7.9M
VIEWS
by http://pornhub.com
collection
eye 7.9M
ameblo.jp
ameblo.jp
collection
135
ITEMS
8.5M
VIEWS
by http://ameblo.jp
collection
eye 8.5M
larepublica.pe
larepublica.pe
collection
1,434
ITEMS
5.5M
VIEWS
by http://larepublica.pe/
collection
eye 5.5M
aol.com
aol.com
collection
789
ITEMS
5.4M
VIEWS
by http://aol.com
collection
eye 5.4M
samsung.com
samsung.com
collection
191
ITEMS
5M
VIEWS
by samsung.com
collection
eye 5M
sourceforge.net
sourceforge.net
collection
210
ITEMS
6.8M
VIEWS
by sourceforge.net
collection
eye 6.8M
elespectador.com
elespectador.com
collection
236
ITEMS
2.7M
VIEWS
by http://elespectador.com/
collection
eye 2.7M
wunderground.com
wunderground.com
collection
253
ITEMS
7.1M
VIEWS
by http://wunderground.com/
collection
eye 7.1M
answers.com
answers.com
collection
100
ITEMS
9.3M
VIEWS
by answers.com
collection
eye 9.3M
thetimes.co.uk
thetimes.co.uk
collection
320
ITEMS
2.7M
VIEWS
by http://thetimes.co.uk
collection
eye 2.7M
nesdis.noaa.gov
nesdis.noaa.gov
collection
135
ITEMS
5.4M
VIEWS
by https://www.nesdis.noaa.gov
collection
eye 5.4M
globalpost.com
globalpost.com
collection
476
ITEMS
4.2M
VIEWS
by http://globalpost.com/
collection
eye 4.2M
slate.com
slate.com
collection
1,534
ITEMS
6.3M
VIEWS
by http://slate.com/
collection
eye 6.3M
nzherald.co.nz
nzherald.co.nz
collection
1,712
ITEMS
5.9M
VIEWS
by http://nzherald.co.nz/
collection
eye 5.9M
yandex.ru
yandex.ru
collection
183
ITEMS
3.8M
VIEWS
by http://yandex.ru
collection
eye 3.8M
macleans.ca
macleans.ca
collection
309
ITEMS
7M
VIEWS
by http://macleans.ca/
collection
eye 7M
imdb.com
imdb.com
collection
2,781
ITEMS
5.7M
VIEWS
by http://imdb.com
collection
eye 5.7M
europe.wsj.com
europe.wsj.com
collection
498
ITEMS
6.1M
VIEWS
by http://europe.wsj.com/
collection
eye 6.1M
livejournal.com
livejournal.com
collection
139
ITEMS
7.9M
VIEWS
by http://livejournal.com
collection
eye 7.9M
rtve.es
rtve.es
collection
2,522
ITEMS
3.9M
VIEWS
by http://rtve.es/noticias/
collection
eye 3.9M
bhaskar.com
bhaskar.com
collection
815
ITEMS
3.4M
VIEWS
by http://bhaskar.com/
collection
eye 3.4M
pressassociation.com
pressassociation.com
collection
685
ITEMS
4.1M
VIEWS
by http://pressassociation.com/
collection
eye 4.1M
theglobeandmail.com
theglobeandmail.com
collection
341
ITEMS
6.3M
VIEWS
by http://theglobeandmail.com/
collection
eye 6.3M